Power Mean Pyramid Scores for Summarization Evaluation
نویسندگان
چکیده
We present Power Mean Pyramid Scores (PMP), an evaluation metric that extends the Pyramid evaluation scheme for summarization by combining Sentence Content Units (SCU) scores using Power Mean. The Pyramid method generates a summarization score by linearly combining component SCU scores. We find that by combining SCU scores using Power Mean, we can optimize a single parameter, α, leading to significantly improved correlation with human judgements. We demonstrate this result through an empirical study based on TAC-08 evaluation.
منابع مشابه
Supervised Learning of Automatic Pyramid for Optimization-Based Multi-Document Summarization
We present a new supervised framework that learns to estimate automatic Pyramid scores and uses them for optimizationbased extractive multi-document summarization. For learning automatic Pyramid scores, we developed a method for automatic training data generation which is based on a genetic algorithm using automatic Pyramid as the fitness function. Our experimental evaluation shows that our new...
متن کاملRevisiting Summarization Evaluation for Scientific Articles
Evaluation of text summarization approaches have been mostly based on metrics that measure similarities of system generated summaries with a set of human written gold-standard summaries. The most widely used metric in summarization evaluation has been the ROUGE family. ROUGE solely relies on lexical overlaps between the terms and phrases in the sentences; therefore, in cases of terminology vari...
متن کاملUsing the pyramid method to create gold standards for evaluation of extraction based text summarization techniques
We present results from using a version of the pyramid method to create gold standards for evaluation of automatic text summarization techniques in the domain of governmental texts. Our results show that the pyramid method may be useful to create gold standards for extraction based summarization techniques using only five human summarisers.
متن کاملMicrosoft Research at DUC2006: Task-Focused Summarization with Sentence Simplification and Lexical Expansion
Our DUC2006 system comprised three main components: a task-focused extractive summarization system, sentence simplification, and lexical expansion of topic words. This paper details each of these components, together with experiments designed to quantify their individual contributions. We include an analysis of our results according to two independent human evaluation methods, the NIST evaluati...
متن کاملBeyond SumBasic: Task-focused summarization with sentence simplification and lexical expansion
In recent years, there has been increased interest in topic-focused multi-document summarization. In this task, automatic summaries are produced in response to a specific information request, or topic, stated by the user. The system we have designed to accomplish this task comprises four main components: a generic extractive summarization system, a topic-focusing component, sentence simplificat...
متن کامل